Image processing system, relay server, and non-transitory computer-readable storage medium

ABSTRACT

A relay server that relays a service provided by a service server transmits information about a screen to be displayed on a web browser of an image processing apparatus to the image processing apparatus, determines a display mode during execution of a job relating to the service by a local application of the image processing apparatus, and transmits the job and information about the determined display mode to the image processing apparatus.

BACKGROUND Field of the Disclosure

The present disclosure relates to an image processing system, a relayserver, and a non-transitory computer-readable storage medium for makingit easy for a user to identify whether a job is a job input from a cloudservice or a job input from a source other than the cloud service.

Description of the Related Art

Multi-function peripherals (MFPs) have been available with a function ofprinting electronic data on sheets and a function of reading (scanning)paper documents and generating electronic data, at low prices. Such MFPshave been widely used not only in offices but also in homes. Some MFPsfurther include a function of connecting to networks and/or provide userinterfaces (UIs) for displaying an operation screen on a liquid crystalpanel to receive user operations.

Japanese Patent Application Laid-Open No. 2019-165427 discusses amulti-function peripheral (MFP) in which a vendor application developedby a third vendor is installed. The vendor application calls afunction-providing application for configuring settings of a printfunction of the MFP via an application programming interface (API) whenthe vendor application uses the print function of the MFP. Furthermore,the vendor application notifies the function-providing application ofuser interface (UI) parameters so that a UI displayed by the vendorapplication and a UI displayed by the function-providing applicationcalled via the API are displayed in a common display form.

SUMMARY

According to an aspect of the present disclosure, an image processingsystem includes a relay server configured to relay a service provided bya service server, and an image processing apparatus including a localapplication configured to control execution of a job received from therelay server. The relay server and the image processing apparatus areconnected together via a network. The relay server includes a firstnetwork communication unit, one or more memories, and one or moreprocessors. The one or more processors and the one or more memories ofthe relay server are configured to cause the relay server to transmit,to the image processing apparatus, information about a screen to bedisplayed on a web browser of the image processing apparatus; determinea display mode during execution of the job; and transmit, to the imageprocessing apparatus, the job and information about the display modedetermined. The image processing apparatus includes a second networkcommunication unit, one or more memories, and one or more processors.The one or more processors and the one or more memories of the imageprocessing apparatus are configured to cause the image processingapparatus to implement the web browser configured to control the displayof the screen based on the information about the screen to be displayedon the web browser transmitted by the relay server, and the localapplication configured to control the display of the screen duringexecution of the job based on the job and the information about thedisplay mode transmitted by the relay server.

Further features of various embodiments will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an example of a configuration of aservice coordination system.

FIG. 2 is a block diagram illustrating a schematic configuration of amulti-function peripheral (MFP).

FIG. 3 is a block diagram illustrating a schematic configuration of aclient apparatus, a relay server, and a service server.

FIG. 4 is a block diagram illustrating a functional configuration of therelay server.

FIG. 5 illustrates an example of an application list screen.

FIG. 6 illustrates an example of a function selection screen after anapplication is activated.

FIG. 7 is a flowchart illustrating a branch determination process for ascreen display after an application is activated.

FIG. 8 is a sequence diagram illustrating a sequence of performing userauthentication and authorization to acquire data of the service server.

FIG. 9 illustrates an example of a folder list screen.

FIG. 10 illustrates an example of a scan settings screen.

FIG. 11 is a sequence diagram illustrating a process of uploading scandata.

FIG. 12 illustrates an example of scan job parameter information.

FIG. 13 illustrates an example of a print settings screen.

FIG. 14 is a sequence diagram illustrating a printing process.

FIG. 15 is a diagram illustrating display modes displayed by a browseapplication.

FIG. 16 is a diagram illustrating display modes displayed for eachmodel.

FIG. 17 is a sequence diagram illustrating a processing flow ofdisplaying a screen including a display mode determination process.

FIG. 18 is a flowchart illustrating a processing flow of the displaymode determination process.

FIG. 19 illustrates an example of a change of a screen display form of afunction selection screen.

FIG. 20 illustrates an example of a dialog screen.

FIG. 21 is a flowchart illustrating a processing flow of changing ascreen display form of a screen provided by a local application of theMFP in executing a scan job.

FIG. 22 illustrates an example of parameter information transmitted fromthe relay server to the MFP in executing a scan job.

FIG. 23 illustrates an example of a screen provided by the localapplication of the MFP in executing a scan job.

FIG. 24 is a flowchart illustrating a processing flow of changing ascreen display form of a screen provided by the local application of theMFP in executing a print job.

FIG. 25 illustrates an example of a screen provided by the localapplication of the MFP in executing a print job.

FIG. 26 illustrates an example of a screen that displays status of jobsbeing executed.

FIG. 27 illustrates an example of a display in a case where status ofjobs being executed is displayed for each display mode.

FIG. 28 illustrates a screen example for displaying a job log.

FIG. 29 illustrates an example of a display in a case where job logs aredisplayed for each display mode.

DESCRIPTION OF THE EMBODIMENTS

There is a multi-function printer (MFP) that includes a web browser andhas a function of displaying screens provided by web servers using theweb browser. With such an MFP, a user operates a user interface (UI)screen provided by a server, and connects the MFP to a cloud service sothat jobs can be input from the cloud service to the MFP.

In examples according to exemplary embodiments described below, an MFPhaving received a job from a cloud service processes the job using alocal application installed in advance in the MFP. As described below,the local application displays a job processing status on a screen suchthat the user can easily identify whether the job being processed is ajob that has been input from the cloud service or a job that has beeninput from a source other than the cloud service.

FIG. 1 is a diagram illustrating a configuration of a servicecoordination system including a relay server according to a firstexemplary embodiment. In the service coordination system illustrated inFIG. 1 , a client apparatus 101, an MFP 102, a relay server 104, and aservice server 106 are connected together via a network 103.

The network 103 is, for example, a local area network (LAN), a wide areanetwork (WAN), a telephone line, or a dedicated digital line. Further,the network 103 can be an asynchronous transfer mode (ATM) or framerelay circuit, a cable television line, or a wireless line for databroadcasting. Alternatively, the network 103 can be a so-calledcommunication network realized by a combination thereof, such as theInternet.

The client apparatus 101 and the MFP 102 are connected to the network103 (Internet) via an access point. The client apparatus 101 and the MFP102 each include a web browser and display webpages provided by a browseapplication 105 of the relay server 104 using the web browser, so that auser can use web services.

The relay server 104 relays data transmission/reception when accessed bythe client apparatus 101 and the MFP 102 so that services provided by aservice application 107 (described below) can be used.

The relay server 104 includes the browse application 105, and the browseapplication 105 manages browsing of webpages by the client apparatus 101and the MFP 102. Hereinafter, the client apparatus 101 and the MFP 102will be referred to collectively as “client terminal”. While the webbrowser installed in the client terminal displays webpages so that theuser can browse the webpages as described below, a description “theclient terminal displays webpages” is sometimes used for asimplification purpose in the present specification.

In the relay server 104, the browse application 105 realizes thefunction of managing browsing of webpages performed by the user usingthe web browser. The browse application 105 transmits predeterminedwebpages when accessed by the client apparatus 101 or the MFP 102. Thepredetermined webpages include webpages relating to various servicesprovided by the service server 106. Specifically, in an exampleaccording to the present exemplary embodiment described herein, theclient terminal does not acquire the webpages relating to various typesof information provided by the service server 106 directly from theservice server 106, but acquires the webpages through the browseapplication 105 to display the acquired webpages. Specifically, theclient terminal displays the webpages provided by the browse application105, so that information about various services provided by the serviceserver 106 is indirectly provided to the user for browsing. For example,the client terminal displays information about a list of files managedby the service server 106 and information about a list of print jobs onwebpages provided by the browse application 105 so that the user canbrowse the displayed information. The predetermined webpages alsoinclude a service login page for logging in to the service server 106from the client terminal. The service login page is a webpage providedby the service server 106.

The service server 106 includes the service application 107. The serviceapplication 107 provides predetermined services to the client apparatus101 and the MFP 102 via the relay server 104. Hereinafter, an“application for providing the predetermined services” will be referredto as a “service application”. On the client apparatus 101 and the MFP102, services provided by the service application 107 are used via therelay server 104. According to the present exemplary embodiment, theservice application 107 manages, for example, image data and documentfile data of the user on the service server 106. The service application107 further provides services for browsing the files on web browsers orother graphical user interfaces (GUIs). Alternatively, the serviceapplication 107 provides services for receiving image data and documentfiles uploaded to the service server 106 via the relay server 104 ordirectly. The service application 107 also provides services forproviding and updating file information via application programminginterfaces (APIs).

Services provided by the service application 107 are not limited tothose described above. The service application 107 can provide variouspublicly-known services. To use the service application 107, it isrequired to log in using pre-registered account information. Accordingto the present exemplary embodiment, the login refers to a process ofentering general account information, authenticating, and acquiringauthentication information, and commonly-known Open Authorization(OAuth) can be used. Alternatively, a method by entering a personalidentification number (PIN) code issued by the service application 107on a terminal of the client apparatus 101 or the MFP 102 can be used.

Any other method by which pre-registered accounts can be identified canalso be used.

According to the present exemplary embodiment, the browse application105 of the relay server 104 transmits a webpage to the client apparatus101 or the MFP 102, and thereby the client terminal (the clientapparatus 101 or the MFP 102) displays the webpage. Thus, the user canbrowse the webpage displayed on the client terminal. The browseapplication 105 executes login processing to log in to the serviceapplication 107 on receipt of a user operation on a displayed webpagefor service login on the client apparatus 101 or the MFP 102. The browseapplication 105 then acquires information about services provided by theservice application 107 using authentication information after the loginprocessing, generates webpages for using the services, and transmits thegenerated webpages to the client apparatus 101 or the MFP 102. In theservice coordination system in FIG. 1 as described above, the relayserver 104 including the browse application 105 and the service server106 including the service application 107 communicate with each otherand transmit results thereof to the client apparatus 101 and the MFP102.

The client apparatus 101 can be any terminal, such as a smartphone, atablet, or a general-purpose personal computer, in which web browserscan be installed.

FIG. 2 is a block diagram illustrating a schematic configuration of theMFP 102 serving as an image processing apparatus. While MFPs are mainlydescribed as an example of peripheral apparatuses in the presentspecification, any type can be used as long as the peripheral apparatusis an image processing apparatuses, such as a copy machine, a facsimile,or a printing apparatus, that can print on paper medium, or an imageprocessing apparatus that includes a scanner unit to scan a paper mediumand converts the scanning results into data.

A print function of the MFP 102 is realized by a printer unit 201. Ascanner function of the MFP 102 is realized by a scanner unit 202. Astorage function of the MFP 102 is realized by a memory card attachmentunit 203 and a memory card 204.

The printer unit 201 prints image data received from external sources orimage data stored in the memory card 204 on print sheets using arecording method, such as an inkjet method or an electrophotographicmethod. The printer unit 201 manages ink information including inklevels and sheet information including the number of stacked sheets.

The scanner unit 202 optically reads an original placed on a platenglass (not illustrated), converts the reading result into electronicdata, converts the electronic data into image data in a designated fileformat, and transmits the image data to an external apparatus via thenetwork 103 or stores the image data in a storage area (notillustrated), such as a hard disk drive (HDD). Further, a copy serviceis realized by transferring the image data generated by reading theoriginal placed on the platen glass using the scanner unit 202 to theprinter unit 201 and printing the transferred image data on a printsheet using the printer unit 201.

The memory card 204 attached to the memory card attachment unit 203stores various types of file data. The file data can be read from anexternal apparatus via the network 103 and edited. File data can also bestored in the memory card 204 from the external apparatus.

The MFP 102 further includes a central processing unit (CPU) 205, aprogram memory 206, a work memory 207, a display unit 208, an operationunit 209, a network communication unit 210, a network connection unit211, and a flash memory 212. The CPU 205 is a central processing unitfor controlling each component in the MFP 102. The program memory 206includes a read-only memory (ROM) and stores various program codes andan application 214 for communicating with a server apparatus. Theapplication 214 includes an application for accessing the printer unit201 and acquiring consumables information about inks and sheets, and anapplication for receiving and executing jobs from a web browser or therelay server 104. The work memory 207 includes a random access memory(RAM) and temporarily stores or buffers image data during serviceexecution. The display unit 208 includes, for example, a liquid crystaldisplay (LCD) and displays various types of information. The operationunit 209 includes switches to be used by the user to execute variousinput operations. The network communication unit 210 connects the MFP102 to the network 103 and executes various types of communication.Connections to the network 103, such as the Internet, can be establishedvia the network connection unit 211. The network communication unit 210supports wired LANs and wireless LANs, and the network connection unit211 for wired LANs is a connector for connecting a wired LAN cable,whereas the network connection unit 211 for wireless LANs is an antenna.The flash memory 212 is a non-volatile memory for storing image datareceived by the network communication unit 210. The foregoing componentsare connected to each other via a bus 213.

FIG. 3 is a block diagram illustrating an example of a hardwareconfiguration of an apparatus that can be used as the client apparatus101, the relay server 104, and the service server 106.

A CPU 301 is a central processing unit for controlling the componentsdescribed below. A disk apparatus 302 stores an application program 312,a database 313, and an operating system (OS) to be read by the CPU 301and also stores various files. The application program 312 is a programfor executing processing for the corresponding apparatus, and details ofthe processing to be executed for the client apparatus 101, details ofthe processing to be executed for the relay server 104, and details ofthe processing to be executed for the service server 106 are differentfrom each other. An external storage medium reading apparatus 303 is anapparatus for reading information, such as files stored in an externalstorage medium like a secure digital (SD) card. A memory 304 is composedof a RAM or the like, and the CPU 301 temporarily stores or buffers datain the memory 304 as needed. A display unit 305 is composed of, forexample, an LCD, to display various types of information. An operationunit 306 includes a keyboard and a mouse to be used by the user toperform various input operations. A network communication unit 307connects to the network 103, such as the Internet, via a networkconnection unit 308 and executes various types of communication. Thenetwork communication unit 307 supports wired LANs and wireless LANs.The network connection unit 308 to be used for a wired LAN is aconnector for connecting a wired LAN cable, whereas the networkconnection unit 308 for a wireless LAN is an antenna. A universal serialbus (USB) communication unit 310 connects to various peripheralapparatuses via a USB connection unit 311 and executes various types ofcommunication. The foregoing components are connected to each other viaa bus 309. It is to be noted that the client apparatus 101, the relayserver 104, and the service server 106 can be a virtual computerrealized on another computer.

A functional configuration will now be described of the relay server 104realized by activation of the browse application 105. FIG. 4 is anexplanatory diagram illustrating a functional configuration of the relayserver 104 in a case where the browse application 105 is activated. Therelay server 104 includes the following functional configuration by thebrowse application 105. The relay server 104 includes atransmission/reception unit 401. The transmission/reception unit 401generates webpages to be transmitted to the web browser of the clientterminal, and transmits and receives various types of informationincluding the webpages to the web browser of the client apparatus 101.The relay server 104 also includes a communication unit 402 and a cookiemanagement unit 403. The communication unit 402 communicates with theservice server 106. The cookie management unit 403 writes, reads, anddeletes cookies of the web browser of the client apparatus 101. Thecookie management unit 403 can encrypt and decrypt values in writing andreading cookies. The relay server 104 further includes a session IDissuing unit 404, a session management unit 405, and a sessioninformation management unit 406. The session ID issuing unit 404 issuesa session identifier (hereinafter, referred to as “session ID”) foruniquely identifying a session that is started in a case where the relayserver 104 is accessed for the first time after the web browser of theclient terminal is activated. The session management unit 405 checkswhether the session ID to be transmitted to the relay server 104 isvalid in writing, reading, and deleting the session ID to or from atemporary area of the web browser of the client terminal or in accessingthe relay server 104 by the web browser of the client terminal. Thesession information management unit 406 manages information inassociation with the session ID. According to the present exemplaryembodiment, the session management unit 405 manages the session ID basedon cookies of the web browser of the client terminal using the cookiemanagement unit 403.

First, processes in a case where the MFP 102 executes printing orscanning will now be described.

FIG. 5 illustrate an example of a screen of an application list 500 thatis displayed on the display unit 305 or 208 in a case where the webbrowser of the client apparatus 101 or the web browser of the MFP 102accesses the browse application 105 of the relay server 104. Theapplication list 500 displays four types of service names 501, 502, 503,and 504 as options. The displayed service names 501, 502, 503, and 504each indicate its corresponding service provided by the service server106. A plurality of services (e.g., service names 501 and 503) can berealized by the same service server or can be provided by a differentservice server from each other. Alternatively, a single server caninclude the functions of the relay server 104 and the service server 106for some or all of the services. Each mark at the left side of a servicename represents an icon from which the service name can be identified asan image. When a MANAGE JOBS button 505 is pressed, a job managementscreen for checking incomplete jobs and executed jobs is displayed.

In coordination with the services, the relay server 104 realizes anapplication function of printing documents or images acquired from theservices using the MFP 102 or transmitting documents or images scannedby the MFP 102 to the services. Possible printing applications are anapplication that prints documents or images stored on the service server106 providing storage services and image sharing services and a printjob execution application that displays and executes print jobs managedin the service server 106. Further, possible scanning applications are astorage application and a profile execution application. The storageapplication stores scanning results on the service server 106 providingstorage services. The profile execution application processes scanningresults based on processing methods registered in the service server106. Each application can include both print and scan functions or canbe configured to execute only one of the print and scan functions.

On the application list 500, only the applications that can be used bythe user with the MFP 102 are desirably selectable. For example, a scanservice 1 (503) is controlled to be disabled or hidden in a case wherethe user is prohibited to use the scan function of the MFP 102 or theMFP 102 includes no scan function.

In a case where an application that provides services corresponding toboth print and scan functions is selected, a screen that prompts theuser to further select a function to use can be displayed as illustratedin FIG. 6 after the application is selected so that the user can use theselected function. In this case, only the functions that can be used aredesirably controlled to be selectable as in the application list 500described above. For example, in a case where the user is prohibited touse the scan function of the MFP 102 or the MFP 102 includes no scanfunction, a scan menu 602 is controlled to be disabled or hidden. In acase where the scan menu 602 is hidden, only a print menu 601 isselectable. Thus, the display of a function selection screen 600 can beskipped, and a screen in a state after the print menu 601 is selectedcan be displayed.

FIG. 7 illustrates a branch determination process that is executed bythe browse application 105 of the relay server 104 after the applicationlist 500 is displayed.

In step S701, if selection of an application by a user operation on theapplication list 500 via the operation unit 306 or 209 is received,information indicating the selected application is transmitted from theclient terminal to the browse application 105 of the relay server 104.

In step S702, the browse application 105 determines whether the selectedapplication is an application that supports both printing and scanning.In a case where the selected application is an application that supportsboth printing and scanning (YES in step S702), the processing proceedsto step S707, whereas in a case where the selected application is anapplication that supports only one of printing and scanning (NO in stepS702), the processing proceeds to step S703. In step S703, the browseapplication 105 determines whether the selected application is aprinting application. In a case where the selected application is aprinting application (YES in step S703), the processing proceeds to stepS710, whereas in a case where the selected application is not a scanningapplication (NO in step S703), the processing proceeds to step S704.

Steps S704 to S706 are a processing flow up to execution of scanning.Specifically, in step S704, the browse application 105 generates afolder selection screen based on the services provided by the serviceserver 106, transmits information about the generated folder selectionscreen to the MFP 102, and displays the folder selection screen on theweb browser of the MFP 102 to prompt the user to select a folder.Thereafter, in step S705, the browse application 105 generates a scansettings screen, transmits the generated scan settings screen to the MFP102, and displays the scan settings screen on the web browser of the MFP102 to prompt the user to configure scan settings. In step S706, in acase where a scan execution instruction is issued, a scan instruction istransmitted together with information about the transmission destinationfolder and the scan settings to the local application of the MFP 102.Consequently, the MFP 102 executes scan processing. In theabove-described example, a folder is selected in step S704. In a case ofa service that executes scan data based on a preset profile, however,the user is prompted to select a scan profile to determine a storagedestination in step S704.

Steps S710 to S712 are a processing flow up to execution of printing. Instep S710, the browse application 105 generates a file selection screenbased on the services provided by the service server 106, transmitsinformation about the generated file selection screen to the MFP 102,and displays the file selection screen on the web browser of the MFP 102to prompt the user to select a print file. In step S711, the browseapplication 105 generates a print settings screen, transmits thegenerated print settings screen to the MFP 102, and causes the MFP 102to display the print settings screen on the web browser of the MFP 102to prompt the user to configure print settings. In step S712, if a printexecution instruction is issued, a print instruction is transmittedtogether with information about the selected file and information aboutthe print settings, to the local application of the MFP 102. The MFP 102thereby executes print processing.

In a case where the selected application is an application that includesboth print and scan functions (YES in step S702), the processingproceeds to step S707. In step S707, it is determined whether the useris being permitted to use both print and scan functions. Thisdetermination is performed using, for example, information about whetherthe MFP 102 being used includes both print and scan functions, orinformation about whether the user as an account of the service server106 is permitted to use both print and scan functions. Whether to permitthe user to use both print and scan functions can be preset by anadministrator of the account. In a case where the user is permitted touse only one of the functions (NO in step S707), the processing proceedsto step S713. In step S713, whether the user is permitted to use theprint function or the scan function is determined. In a case where theuser is permitted to use only the print function (YES in step S713), theprocessing flow of printing from step S710 described above is executed.Whereas in a case where the user is permitted to use only the scanfunction (NO in step S713), the processing flow of scanning from stepS704 described above is executed. In contrast, in step S707, in a casewhere the user is determined as being permitted to use both print andscan functions (YES in step S707), the processing proceeds to step S708.In step S708, the print/scan function selection screen illustrated inFIG. 6 is displayed to prompt the user to select a desired function. Ina case where it is determined that the print function is selected (YESin step S709), the processing flow of printing from step S710 describedabove is executed, whereas in a case where it is determined that thescan function is selected (NO in step S709), the processing flow ofscanning from step S704 described above is executed.

FIG. 8 illustrates a sequence diagram from the user authentication andauthorization to the display of a screen for selecting a list of foldersof the user or a list of print jobs in a case where the web services ofboth the service server 106 and the relay server 104 coordinate togetherand realize file acquisition and uploading processing. In here,commonly-known OAuth will be described as an example. In OAuth, therelay server 104 plays a role of a client, and the service server 106plays a role of a resource server. While a case is described hereinwhere the user uses the browse application 105 of the relay server 104using the display unit 208 and the operation unit 209 of the MFP 102,similar processing and operations are executed also in a case where theuser uses the client apparatus 101.

When, in step S801, an application is selected from the application list500 via the operation unit 209 of the MFP 102, the MFP 102 transmits arequest to activate the selected application to the relay server 104. Instep S802, the relay server 104 then checks an authentication statestored on the service server 106.

In a case where, in step S802, the service server 106 determines thatthe authentication state is invalid, the relay server 104 transmits, instep S803 to the MFP 102, a redirect request to redirect to a uniformresource locator (URL) for authentication by the service server 106.This corresponds to a case where authentication information in theprevious authentication has been expired or a case where the MFP 102 isattempting to access the service server 106 for the first time.

In step S804, the MFP 102 accesses the URL for authentication by theservice server 106 based on the received redirect request. In step S805,the MFP 102 displays an authentication screen on the display unit 208 ofthe MFP 102.

The MFP 102 acquires information (e.g., account ID, password) forauthentication from the user via the authentication screen, andtransmits, in step S806, the received information to the service server106. In step S807, the service server 106 thereby transmits a callbackrequest after completion of authentication to the MFP 102.

In response to the callback request, the MFP 102 transmits, in stepS808, callback information to the relay server 104. In a case where theauthenticated user has not completed authorization processing for usingthe functions provided by the service server 106, the processing flow ofsteps S805 and S806 is executed again to redirect to an authorizationscreen, to display a permission screen to acquire permissions fornecessary items from the user, and to transmit information about thepermissions from the user to the service server 106.

In a case where the authentication screen displayed in step S805 is in aform that cannot be displayed on the display unit 208 of the MFP 102,the authentication and authorization operations in steps S805 to S807can be executed on another terminal, such as the client apparatus 101.

When the authentication and authorization operations are completednormally and the relay server 104 receives the callback information, therelay server 104 acquires, in step S809, necessary access tokens foracquiring authenticated account information from the service server 106based on the received callback information. In step S810, the relayserver 104 acquires, using the acquired access tokens, folderinformation and a print job list of the service server 106 associatedwith the authenticated account. In step S811, the relay server 104returns the received list to the MFP 102 and generates a folder/filedisplay screen or a print job display screen on the display unit 208 ofthe MFP 102. FIG. 9 illustrates an example of a folder/file displayscreen.

In FIG. 9 , folders 901 and 902 indicate folders of the user on theservice server 106. Files 903, 904, and 905 indicate files that are onthe same hierarchical layer as the folders 901 and 902.

When a scanning application is executed or a scan function of anapplication capable of printing and scanning is executed, the user isprompted to select a scanning result uploading destination folder byoperating the operation unit 209. In this case, the file 903, the file904, and the file 905 are not selectable. Alternatively, the files thatcannot be selected can be hidden.

When any one of the folders 901 and 902 is selected, the MFP 102transmits information about the selected folder to the relay server 104.Consequently, steps S810 and S811 are executed again, and a folder/filedisplay screen 900 is updated to display sub-folders and files stored inthe selected folder. If a SELECT button 906 is pushed, any folder thatis displayed at the time of the selection is selected as an uploadingdestination folder. In the state illustrated in FIG. 9 , the folderincluding the folders 901 and 902 and the files 903 to 905 is selected.

When a printing application is executed or a print function of anapplication capable of printing and scanning is executed, the user isprompted to select a printing target file by operating the operationunit 209. Even at this time, sub-folders and files stored in theselected folder are displayed as in executing the scan functiondescribed above in a case where a folder 901 or a folder 902 isselected. In executing the print function, a file to be printed needs tobe identified. Thus, the SELECT button 906 is enabled only in a statewhere a file is being selected.

FIG. 10 illustrates an example of a screen for configuring the scansettings after the SELECT button 906 is pushed in executing the scanfunction. Possible items of settings that can be changed are typically atype of an original 1001, a sheet size 1002, an orientation of theoriginal 1003, and a resolution 1004. However, any item that is settablefor the MFP 102 can be set, or all items can be predetermined andnothing is to be set in executing scanning.

When an EXECUTE SCANNING button 1005 is selected, the MFP 102 transmitsa notification indicating that the EXECUTE SCANNING button 1005 isselected to the relay server 104. Consequently, a scan data uploadingprocess illustrated in FIG. 11 is started.

In step S1101, the relay server 104 transmits a scan instruction to theMFP 102 using parameters of the scan settings configured by the user viaa scan settings screen 1000.

The scan instruction can include other parameters (e.g., UI display forminformation) specifying behaviors in a case where the MFP 102 executesthe scan job together with the parameters of the scan settings. FIG. 12illustrate an example of parameters 1200 (hereinafter, referred to as“scan parameters”) transmitted from the relay server 104 to the MFP 102in transmitting the scan instruction (step S1101). In the example of thescan parameters, the parameters of the scan settings described above(tickets 1206) and other parameters including a job ID 1201, a job type1202, a job name 1203, an application ID 1204 of the application bywhich the job has been input, and job input date and time 1205 aredescribed.

In step S1102, the MFP 102 having received the scan instruction executesscanning based on the designated parameters of the scan settings. Instep S1103, the MFP 102 transmits scanning result data to the relayserver 104. While the scanning is being executed (step S1102), a screenindicating that scanning is being executed can be displayed on thedisplay unit 208. Furthermore, in a case where the parameters specifyingbehaviors are received from the relay server 104, behaviors (e.g.,display form) of the display unit 208 can be changed based on theparameters. In step S1104, the relay server 104 uploads binary data onthe received scanning result to the service server 106, which is astorage server associated with the account, using the access tokensacquired in step S809 in FIG. 8 . The folder information selected by theuser via the folder/file display screen 900 in FIG. 9 is alsotransmitted, and an instruction to store the uploaded scan data in thefolder selected by the user is issued.

FIG. 13 illustrate an example of a screen for configuring the printsettings after the SELECT button 906 is selected in executing the printfunction. While possible items of settings that can be set are typicallya sheet size 1301, a print layout 1302, a 2-sided setting 1303, and acolor mode 1304, any items that are settable for the MFP 102 can be set.Alternatively, all items can be predetermined and nothing is to be setin executing printing.

When an EXECUTE PRINTING button 1305 is pushed, the MFP 102 transmits anotification of information indicating that the EXECUTE PRINTING button1305 has been pushed to the relay server 104. Consequently, a printingprocess illustrated in FIG. 14 is started.

In step S1401, the relay server 104 transmits a request to downloadprint data for executing printing to the service server 106. In stepS1402, when the download of the print data is completed, the relayserver 104 transmits, to the MFP 102, a print instruction to executeprinting using the parameters of the print settings configured via aprint settings screen 1300 and the print data downloaded in step S1401.The print instruction can include, together with the parameters of theprint settings, parameters (e.g., UI display form information)specifying behaviors in a case where the MFP 102 executes the print job.

In step S1403, the MFP 102 having received the print instructionexecutes printing based on the designated parameters of the printsettings. While the printing is being executed (step S1403), a screenindicating that printing is being executed can be displayed on thedisplay unit 208. Furthermore, in a case where the parameters specifyingbehaviors are received from the relay server 104, behaviors (e.g.,display form) of the display unit 208 can be changed based on theparameters. In step S1404, after completing the printing, the MFP 102transmits, to the relay server 104, a printing result indicating thatthe printing has been completed successfully, a printing resultindicating that an error has occurred, or a printing result indicatingthat the printing has been cancelled by the user. The printing resultnotification can include information from which the consumption ofconsumables can be estimated, such as information about the number ofpages printed on sheets that are used, information indicating which oneof color printing and monochrome printing is executed, and informationindicating whether 2-sided printing is executed. In a case where theprinting is cancelled by the user while the printing is being executed,information about the executed part of the printing is transmitted tothe relay server 104.

Hereinafter, the following items will be described: a method forproviding a screen to be displayed on the client terminal and an exampleof changing a display form of the screen according to the presentexemplary embodiment, and a method for providing a screen to bedisplayed in executing a job by the local application of the MFP 102 andan example of changing a display form of the screen.

First, a method for determining a display mode, a method for providing ascreen to be displayed on the client terminal, and an example ofchanging a display form will now be described with reference to FIGS. 15to 19 .

FIG. 15 illustrates an example of a data structure for managing displaymodes that can be displayed by the browse application 105 for eachservice application. The data structure presents display modes that canbe displayed by each application in tabular form. There may be a casewhere the number of display modes that can be displayed by anapplication is zero, one, or more. In a case where the number of displaymodes that can be displayed is more than one, the display modes can beprioritized. For example, FIG. 15 indicates that a scan service 1 can bedisplayed in two display modes that are a work mode and a study mode.While the tabular form is used in the example, any form of datastructure can be used as long as the data structure can determine adisplayable display mode if an application is given. A service without adisplayable display mode is to be displayed in a default display form,and the changing of display form is not executed.

FIG. 16 illustrates an example of a data structure for managing displaymodes that can be displayed by the MFP 102 for each model identificationID. Each model identification ID is an ID determined uniquely for eachMFP. There may be a case where the number of display modes that can bedisplayed by a model is zero, one, or more. In a case where the numberof display modes that can be displayed is more than one, the displaymodes can be prioritized. For example, FIG. 16 indicates that a model Acan be displayed in the display mode “work mode”. While the tabular formis used in the example, any form can be used of data structure thatdetermines a displayable display mode if a model identification ID isgiven.

FIG. 17 illustrates a sequence diagram in which a screen display requestis issued by the web browser of the client terminal, a screen displaymode to be provided is determined by the relay server 104 havingreceived the screen display request, and a screen is provided to the webbrowser of the client terminal. In step S1701, the client terminaltransmits a screen display request to the relay server 104. When therelay server 104 receives the screen display request from the clientterminal, the relay server 104 determines a screen to be provided to theclient terminal based on the branch determination process illustrated inFIG. 7 and generates screen configuration information for displaying thescreen on the web browser of the client terminal. The screenconfiguration information described herein indicates information thatdefines behaviors of the web browser and includes, for example,HyperText Markup Language (HTML), Cascading Style Sheets (CSS), and URLsfor acquiring HTML or CSS. In step S1702, the relay server 104determines a display mode based on a display mode determination processdescribed below. There may be a case where there is no display mode as aresult of the display mode determination process. In a case where adisplay mode is determined in step S1702, screen configurationinformation with a screen display form changed based on the determineddisplay mode is returned to the client terminal. Which part of thescreen display form is to be changed and details of the change to bemade can be defined for each display mode. For example, completelydifferent display forms can be defined for the work mode and the studymode, or different display forms that are partially the same can bedefined for the work mode and the study mode. In a case where there isno display mode, screen configuration information is returned to theclient terminal without changing the display form. In step S1703, whenthe client terminal receives the screen configuration information fromthe relay server 104, the screen is displayed as a webpage on the webbrowser of the client terminal.

FIG. 18 illustrates the display mode determination process that isexecuted by the relay server 104 when a screen display request istransmitted from the web browser of the client terminal to the relayserver 104. In the display mode determination process (steps S1804 toS1806) described below, there may be a case where setting information isread from the database 313 or acquired from an external server bycommunicating with the external server via the network 103. However,executing the foregoing processing each time the relay server 104receives a screen display request from the client terminal may causedelays in responding to the screen display requests. Thus, a displaymode acquired in advance is stored in the session information managementunit 406 as session information between the web browser of the clientterminal and the relay server 104, and the display mode is desirablyacquired from the session information management unit 406 while thesession between the web browser of the client terminal and the relayserver 104 is valid.

When the relay server 104 receives a screen display request from the webbrowser of the client terminal, the relay server 104 acquires a displaymode through the following determination process. In step S1801, thesession management unit 405 first checks whether the session is validbased on a session ID transmitted from the web browser of the clientterminal. If the session ID is valid (YES in step S1801), the processingproceeds to step S1802. In step S1802, the session management unit 405attempts to acquire a display mode from the session informationmanagement unit 406. In step S1803, it is determined whether a displaymode is successfully acquired from the session information managementunit 406, whether information indicating that there is no display modeis acquired, or whether the attempt to acquire a display mode hasfailed. If a display mode is successfully acquired (YES in step S1803),the acquired display mode is determined, and the display modedetermination process is ended. If information indicating that there isno display mode is acquired, it is determined that there is no displaymode, and the display mode determination process is ended. In a casewhere there is no session ID or the session ID is invalid or in a casewhere the display mode acquisition from the session informationmanagement unit 406 has failed due to expiration of the session, adisplay mode is determined by the following process. In step S1804, therelay server 104 acquires display modes supported by the browseapplication 105 having issued the screen display request. At this time,the relay server 104 acquires display modes supported by the browseapplication 105 using data having the data structure illustrated in FIG.15 . In step S1805, the relay server 104 then acquires display modessupported by the MFP 102. At this time, the relay server 104 acquiresdisplay modes supported by the MFP 102 using data having the datastructure illustrated in FIG. 16 . In the display mode acquisition insteps S1804 and S1805, display modes can be acquired from theapplication program 312 or the database 313 or from an external servervia the network 103. In step S1806, a display mode is determined bycalculating a logical product of the display modes that can be displayedby the selected browse application 105 for the services and the displaymodes that can be displayed by the MFP 102. As illustrated in FIGS. 15and 16 , there is a plurality of display modes that the browseapplication 105 supports for each service and a plurality of displaymodes that the MFP 102 supports. Thus there may be a case where adisplay mode cannot be determined uniquely. In this case, a display modeis determined based on a predetermined priority order. For example, in acase where there is a plurality of display modes that can be displayedby the browse application 105, and the plurality of display modes isprioritized, a display mode can be determined based on the set priorityorder. In a case where there is a plurality of display modes that can bedisplayed by the MFP 102 and the plurality of display modes isprioritized, a display mode can be determined based on the set priorityorder, or the priority order can be written to a program of theapplication 214 or a program of the browse application 105. In a casewhere a display mode is determined as a result of step S1806, in stepS1807, the determined display mode is written to the session informationmanagement unit 406. In a case where there is no display mode as aresult of step S1806, in step S1807, information indicating that thereis no display mode is written to the session information management unit406.

FIG. 19 illustrates an example of a change of a display form of theprint/scan function selection screen 600 in FIG. 6 that the relay server104 provides to the client terminal. A screen 1901 is an example of ascreen as a result of changing a display form of a function selectionscreen 1900, which is a screen before the display form is changed, basedon the display mode determined by the display mode determination processdescribed above. At this time, the display form is changed by increasingthe thickness of an outer frame of the function selection screen from anormal thickness, changing the color of the outer frame, and displayingan application name 1902 on the screen (i.e., displaying characters thatare not displayed on the function selection screen 1900 on the screen),as an example of changing the display form based on the display mode.However, any display form changing method can be used as long as thedisplay form changing method provides a UI from which the user canrecognize that the display mode is different from a normal display mode.

FIG. 20 illustrates an example of a dialog screen that is provided bythe browse application 105 via the web browser of the MFP 102 anddisplayed on the display unit 208 in a case where an exceptionalsituation occurs or a message is to be checked by the user. An outerframe of a dialog screen 2000 is colored even in a case where the screendisplay form is not changed, and the color of the outer frame variesdepending on how important the message is. If the change of the displayform in the example illustrated in FIG. 19 is uniformly applied to thedialog screen 2000, the original intention to indicate the importance ofthe message to the user is no longer achieved. Thus, the change of thedisplay form is not applied exceptionally. As described above, there maybe screens that are not changed in display form for various reasons.

A method for providing a screen to be displayed by the local applicationof the MFP 102 during job execution and a method for changing a displayform of a screen to be displayed on the display unit 208 according tothe present exemplary embodiment will now be described. While a scan joband a print job will be described as examples of job types, any job canbe used as long as the relay server 104 can provide operationinstructions to the MFP 102 in the job. For example, a copy job or a jobof turning on/off the MFP 102 can be used. Further, a method forchanging a display form of a screen to be displayed by the localapplication of the MFP 102 during job execution will be described belowstarting from the press of a button for issuing a job executioninstruction that is displayed on the client terminal (the MFP 102 or theclient apparatus 101). However, any job input method can be used as longas jobs can be input to the MFP 102 via the relay server 104 by usingthe job input method. For example, a method can be used for changing adisplay form of a screen to be displayed by the local application of theMFP 102 during job execution by inputting a job to the MFP 102 via therelay server 104 starting from an automated process triggered by ascreen operation on the service server 106 or by date and time.

FIG. 21 illustrates a sequence diagram from the press of the EXECUTESCANNING button 1005 to the display of the screen in the changed displayform on the display unit 208 during execution of scanning. In stepS2101, if the EXECUTE SCANNING button 1005 is pressed, the relay server104 acquires a display mode based on the display mode determinationconditions illustrated in FIG. 17 . In step S2102, the relay server 104transmits a scan instruction to the MFP 102. At this time, the displaymode acquired in step S2101 is transmitted as part of scan parameterinformation transmitted to the MFP 102 in the scan instruction in stepS2102. FIG. 22 illustrates an example of scan parameters transmitted tothe MFP 102 in the scan instruction in step S2102.

Scan parameters 2200 include a display mode 2201 in addition toparameters described in the example of scan parameters illustrated inFIG. 12 . While the form in which the display mode is transmitted aspart of the job setting information in the scan instruction to the MFP102 is described, any method by which the MFP 102 can acquire thedisplay mode can be used. For example, the MFP 102 can request thedisplay mode from the relay server 104 after receiving the scaninstruction. In step S2103, if the MFP 102 receives the scaninstruction, the MFP 102 executes a scan job. At this time, the MFP 102changes, in step S2104, the display form of the UI to be displayed onthe display unit 208 based on the display mode received from the relayserver 104, and changes and displays the display form of the UI. FIG. 23illustrates an example of a screen in a changed display form. A screen2301 is an example of a screen in a display form that is not changed,whereas a screen 2302 is an example of a screen in a display form thatis changed based on a display mode received from the relay server 104 bythe MFP 102. For example, in a case where the display mode is the workmode, the screen 2302 is displayed in a form with a changed backgroundcolor.

FIG. 24 illustrates a sequence diagram up to the display of a screen ina changed display form on the display unit 208 during execution ofprinting after the EXECUTE PRINTING button 1305 is pressed. In stepS2401, if the EXECUTE PRINTING button 1305 is pressed, the relay server104 acquires a display mode based on the display mode determinationconditions illustrated in FIG. 17 . In step S2402, the relay server 104acquires print data. In step S2403, the relay server 104 then transmitsa print instruction to the MFP 102. At this time, the relay server 104transmits the display mode acquired in step S2401 as part of job settinginformation transmitted to the MFP 102 together with the printinstruction. The form has been described in which the display mode istransmitted as part of the job setting information in the printinstruction to the MFP 102. However, any method by which the MFP 102 canacquire the display mode can be used. For example, the MFP 102 canrequest the display mode from the relay server 104 after receiving theprint instruction. In step S2404, if the MFP 102 receives the printinstruction, the MFP 102 executes a print job. At this time, in stepS2405, the MFP 102 changes and displays the display form of the UI to bedisplayed on the display unit 208 based on the display mode receivedfrom the relay server 104. FIG. 25 illustrates an example of a screen ina changed display form. A screen 2501 is an example of a screen in adisplay form that is not changed, whereas a screen 2502 is an example ofa screen in a display form that is changed based on a display modereceived from the relay server 104 by the MFP 102. For example, in acase where the display mode is the work mode, the screen 2502 isdisplayed in a form with a changed background color.

As described above, the local application of the MFP 102 displays aprocessing progress status on a normal screen, such as the screen 2301in FIG. 23 or the screen 2501 in FIG. 25 , in a case where the localapplication executes scanning or printing by operating a local UI on theMFP 102. On the contrary, in a case where a scan service or a printservice provided by the service server 106 is used via a screen providedby the relay server 104, the local application of the MFP 102 displays ascreen in a changed display form, such as the screen 2302 in FIG. 23 orthe screen 2502 in FIG. 25 . This makes it easy for the user of the MFP102 to identify whether a job being processed by the MFP 102 is a jobbased on a service provided by the service server 106.

The foregoing description describes a method for changing a display formof a screen to be displayed on the web browser of the client terminaland a method for changing a display form of a screen provided by thelocal application during job execution according to the first exemplaryembodiment. By adding a display mode as parameter information during jobexecution, a display form of a screen that displays a job executionstatus list and a display form of a screen that displays an executed joblog information list can also be changed based on the display mode. Amethod for changing a display form of a job management screen accordingto a second exemplary embodiment will now be described.

FIG. 26 illustrates an example of a job status list screen displaying anincomplete job list. If the MANAGE JOBS button 505 is pressed, a jobstatus list screen 2600 displaying a list of incomplete print jobs andincomplete scan jobs is displayed. While the incomplete jobs aredisplayed at the press of the MANAGE JOBS button 505 according to thepresent exemplary embodiment, any procedure for displaying the jobstatus list screen 2600 can be used. Further, while the print jobs andthe scan jobs are listed together, the print jobs can be displayed on ascreen and the scan jobs can be displayed on another screen. While printjobs and scan jobs are described herein as an example, other jobs (e.g.,copy job) can also be displayed. In a case where there are incompletejobs, the incomplete jobs are listed on the job status list screen 2600.In the example illustrated in FIG. 26 , three job names 2601, 2602, and2603 are displayed on the job status list screen 2600. The job name 2601is a print job. The job name 2602 is a scan job. The job name 2603 is aprint job that is input in the work mode. Each job name can be given aname at the time of inputting the job, and a file name of a print filecan be used for the print job name but does not have to be used.Further, job input date/time, a job execution state 2604, and an icon2605 from which the browse application 105 having input the job can beidentified as an image can be displayed together, or the display formcan be changed based on the display mode transmitted as job settinginformation at the time of inputting the job. While a background of thejob name 2603 is changed, any method can be used for changing thedisplay form so that the user can recognize the display mode. Forexample, a color of characters of a job name can be changed.Furthermore, jobs can be grouped for each display mode and displayed bygroup as illustrated in FIG. 27 . In the example in FIG. 27 , a list ofincomplete jobs that are jobs input in the work mode is displayed in ajob status list screen 2700, and at the press of a display modeselection button 2701, the current screen changes to a screen thatdisplays incomplete jobs of another display mode (e.g., study mode).

FIG. 28 illustrates an example of a job log list screen displaying anexecuted job list. At the press of a log list display button 2606, thecurrent screen changes to the job log list screen. While a job log listscreen 2800 is displayed at the press of the log list display button2606 according to the present exemplary embodiment, any procedure fordisplaying the job log list screen 2800 can be used. While the printjobs and the scan jobs are listed together, the print jobs can bedisplayed on a screen and the scan jobs can be displayed on anotherscreen. While print jobs and scan jobs are described herein as anexample, another job (e.g., copy job) can also be displayed. In a casewhere there are executed jobs, the executed jobs are listed on the joblog list screen 2800. In the example illustrated in FIG. 28 , three jobnames 2801, 2802, and 2803 are displayed on the job log list screen2800. The job name 2801 is a scan job. The job name 2802 is a print job.The job name 2803 is a print job that is input in the work mode. As inthe job status list screen 2600, each job name can be given any name atthe time of inputting the job. A file name of a print file can also beused but does not have to be used. Further, job input date/time and anicon 2804 from which the browse application 105 having input the job canbe identified as an image can be displayed together, or the display formcan be changed based on the display mode transmitted as job settinginformation at the time of inputting the jog. While a background of thejob name 2803 is changed, any method for changing the display form canbe used so that the user can recognize the display mode. For example, acolor of characters of a job name can be changed. At the press of aSTATUS LIST button 2805, the current screen changes to the job statuslist screen 2600. As in FIG. 29 , executed jobs can be grouped for eachdisplay mode and displayed by group. In the example illustrated in FIG.29 , a list of executed jobs that are jobs input in the work mode isdisplayed, and at the press of a display mode selection button 2901, thecurrent screen changes to a screen that displays executed jobs ofanother display mode (e.g., study mode). While the screen 2800 thatdisplays an incomplete job list and the screen 2900 that displays anexecuted job log list are separated, incomplete jobs and executed joblogs can be displayed on the same screen.

OTHER EMBODIMENTS

Embodiment(s) of the present disclosure can also be realized by acomputer of a system or apparatus that reads out and executescomputer-executable instructions (e.g., one or more programs) recordedon a storage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer-executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer-executable instructions. The computer-executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present disclosure has described exemplary embodiments, it isto be understood that some embodiments are not limited to the disclosedexemplary embodiments. The scope of the following claims is to beaccorded the broadest interpretation so as to encompass all suchmodifications and equivalent structures and functions.

This application claims priority to Japanese Patent Application No.2022-108148, which was filed on Jul. 5, 2022 and which is herebyincorporated by reference herein in its entirety.

What is claimed is:
 1. An image processing system in which a relayserver configured to relay a service provided by a service server and animage processing apparatus including a local application configured tocontrol execution of a job received from the relay server are connectedtogether via a network, the relay server comprising: a first networkcommunication unit, one or more memories, and one or more processors,wherein the one or more processors and the one or more memories of therelay server are configured to cause the relay server to: transmit, tothe image processing apparatus, information about a screen to bedisplayed on a web browser of the image processing apparatus; determinea display mode during execution of the job; and transmit, to the imageprocessing apparatus, the job and information about the display modedetermined, and the image processing apparatus comprising: a secondnetwork communication unit, one or more memories, and one or moreprocessors, wherein the one or more processors and the one or morememories of the image processing apparatus are configured to cause theimage processing apparatus to implement: the web browser configured tocontrol the display of the screen based on the information about thescreen to be displayed on the web browser transmitted by the relayserver; and the local application configured to control the display ofthe screen during execution of the job based on the job and theinformation about the display mode transmitted by the relay server. 2.The image processing system according to claim 1, wherein the job is aprint job for printing a file.
 3. The image processing system accordingto claim 1, wherein the job is a scan job for executing scanning.
 4. Theimage processing system according to claim 1, wherein the one or moreprocessors and the one or more memories of the image processingapparatus are further configured to cause the image processing apparatusto transmit a result of execution of the job to the relay server.
 5. Theimage processing system according to claim 4, wherein the relay serverfurther comprises a storage unit configured to store a received resultof execution of the job.
 6. The image processing system according toclaim 1, wherein the one or more processors and the one or more memoriesof the relay server are further configured to cause the relay server todetermine the display mode based on at least one of a type of theservice provided by the service server or a type of the image processingapparatus.
 7. A relay server configured to relay a service provided by aservice server, the relay server comprising: a first transmission unitconfigured to transmit, to an image processing apparatus, informationabout a screen to be displayed on a web browser of the image processingapparatus; a determination unit configured to determine a display modeduring execution of a job relating to the service by a local applicationof the image processing apparatus; and a second transmission unitconfigured to transmit, to the image processing apparatus, the job andinformation about the display mode determined.
 8. A method forcontrolling a relay server configured to relay a service provided by aservice server, the method comprising: transmitting, to an imageprocessing apparatus, information about a screen to be displayed on aweb browser of the image processing apparatus; determining a displaymode during execution of a job relating to the service by a localapplication of the image processing apparatus; and transmitting, to theimage processing apparatus, the job and information about the displaymode determined.
 9. A non-transitory computer-readable storage mediumstoring instructions that, when executed by a computer of a relayserver, cause the computer of the relay server to perform: transmitting,to an image processing apparatus, information about a screen to bedisplayed on a web browser of the image processing apparatus;determining a display mode during execution of a job by a localapplication of the image processing apparatus, the job relating to aservice provided by a service server; and transmitting, to the localapplication of the image processing apparatus, the job and informationabout the display mode determined.